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ABSTRACT 

To obtain good system performance, a DBA must choose a set 
of indices tiiat is appropriate for the workload. The system can 
aid in this challenging task by providing recommendations for the 
index configuration. We propose a new index recommendation 
technique, termed semi-automatic tuning, that keeps the DBA "in 
the loop" by generating recommendations that use feedback about 
the DBA's preferences. The technique also works online, which 
avoids the limitations of commercial tools that require the work- 
load to be known in advance. The foundation of our approach is 
the Work Function Algorithm, which can solve a wide variety of 
online optimization problems with strong competitive guarantees. 
We present an experimental analysis that validates the benefits of 
semi-automatic tuning in a wide variety of conditions. 

1 Introduction 

Index tuning, i.e., selecting indices that are appropriate for the 
workload, is a crucial task for database administrators (DBAs). 
However, selecting the right indices is a very difficult optimization 
problem: there exists a very large number of candidate indices for a 
given schema, indices may benefit some parts of the workload and 
also incur maintenance overhead when the data is updated, and the 
benefit or update cost of an index may depend on the existence of 
other indices. Due to this complexity, an administrator often resorts 
to automated tools that can recommend possible index configura- 
tions after performing some type of workload analysis. 

In this paper, we introduce a novel paradigm for index tuning 
tools that we term semi-automatic index tuning. A semi-automatic 
index tuning tool generates index recommendations by analyzing 
the workload online, i.e., in parallel with query processing, which 
allows the recommendations to adapt to shifts in the running work- 
load. The DBA may request a recommendation at any time and is 
responsible for selecting the indices to create or drop. The most 
important and novel feature of semi-automatic tuning is that the 
DBA can provide feedback on the recommendation, which is taken 
into account for subsequent recommendations. In this fashion, the 
DBA can refine the automated recommendations by passing indi- 
rect domain knowledge to the tuning algorithm. Overall, the semi- 
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automatic paradigm offers a unique combination of very desirable 
features: the tuner analyzes the running workload online and thus 
relieves the DBA from the difficult task of selecting a representa- 
tive workload; the DBA retains total control over the performance- 
critical decisions to create or drop indices; and, the feedback mech- 
anism couples human expertise with the computational power of an 
automated tuner to enable an iterative approach to index tuning. 

We illustrate the main features of semi-automatic tuning with 
a simple example. Suppose that the semi-automatic tuner recom- 
mends to materialize three indices, denoted a, b, and c. The DBA 
may materialize a, knowing that it has negligible overhead for the 
current workload. We interpret this as implicit positive feedback 
for a. The DBA might also provide explicit negative feedback on 
c because past experience has shown that it interacts poorly with 
the locking subsystem. In addition, the DBA may provide posi- 
tive feedback for another index d that can benefit the same queries 
as c without the performance problems. Based on this feedback, 
the tuning method can bias its recommendations in favor of indices 
a, d and against index c. For instance, a subsequent recommenda- 
tion could be {a, d, e}, where e is an index that performs well with 
d. At the same time, the tuning method may eventually override the 
DBA's feedback and recommend dropping some of these indices if 
the workload provides evidence that they do not perform well. 

Previous Work. Existing approaches to index selection fall in two 
paradigms, namely offline and online. Offline techniques j2] |6j 
generate a recommendation by analyzing a representative workload 
provided by the DBA, and let the DBA make the final selection 
of indices. However, the DBA is faced with the non-trivial task 
of selecting a good representative workload. This task becomes 
even more challenging in dynamic environments (e.g., ad-hoc data 
analytics) where workload patterns can evolve over time. 

Online techniques |5][T0][l3 14 1 monitor the workload and au- 
tomatically create or drop indices. Online monitoring is essential 
to handle dynamic workloads, and there is less of a burden on the 
DBA since a representative workload is not required. On the other 
hand, the DBA is now completely out of the picture. DBAs are 
typically very careful with changes to a running system, so they are 
unlikely to favor completely automated methods. 

None of the existing index tuning techniques achieves the same 
combination of features as semi-automatic tuning. Semi-automatic 
tuning starts with the best features from the two paradigms (on- 
line workload analysis with decisions delegated to the DBA) and 
augments them with a novel feedback mechanism that enables the 
DBA to interactively refine the recommendations. We note that in- 
teractive index tuning has been explored in the literature (VJ, but 
previous studies have focused on offline workload analysis. Our 
study is the first to propose an online feedback mechanism that is 
tightly coupled with the index recommendation engine. 



A closer look at existing techniques also reveals that they can- 
not easily be modified to be semi-automatic. For instance, a naive 
approach to semi-automatic tuning would simply execute an online 
tuning algorithm in the background and generate recommendations 
based on the current state of the algorithm, but this approach ig- 
nores the fact that the DBA may select indices that contradict the 
recommendation. A key challenge of semi-automatic tuning is to 
adapt the recommendations in a flexible way that balances the in- 
fluence of the workload and feedback from the DBA. 

Our Contributions. We propose the WFIT index-tuning algorithm 
that realizes the new paradigm of semi-automatic tuning. WFIT uses 
a principled framework to generate recommendations that take the 
workload and user feedback into account. We can summarize the 
technical contributions of this paper as follows: 

• We introduce the new paradigm of semi-automatic index tuning 
in Section |3] We identify the relevant design choices, provide a 
formal problem statement, and outline the requirements for an ef- 
fective semi-automatic index advisor. 

• We show that recommendations can be generated in a princi- 
pled manner by an adaptation of the Work Function Algorithm | 3 1 
(WFA) from the study of metrical task systems (Section [4. l| l. We 
prove that WFA selects recommendations with a guaranteed bound 
on worst-case performance, which allows the DBA to put some 
faith in the recommended indices. The proof is interesting in the 
broader context of online optimization, since the index tuning prob- 
lem does not satisfy the assumptions of the original Work Function 
Algorithm for metrical task systems. 



• We develop the WFA+ algorithm (Section 4.2 1 which uses a divide- 



and-conquer strategy with several instances of WFA on separate in- 
dex sets. We show that WFA^ leads to improved running time and 
better guarantees on recommendation quality, compared to analyz- 
ing all indices with a single instance of WFA. The guarantees of 
WFA+ are significantly stronger compared to previous works for 
online database tuning (5||10|, and are thus of interest beyond the 
scope of semi-automatic index selection. 

• We introduce the WFIT index-tuning algorithm that provides an 
end-to-end implementation of the semi-automatic paradigm (Sec- 
tion [sj. The approach builds upon the framework of WFA+, and 
couples it with two additional components: a principled feedback 
mechanism that is tightly integrated with the logic of WFA+, and an 
online algorithm to extract candidate indices from the workload. 

• We evaluate WFIT's empirical performance using a prototype im- 
plementation over IBM DB2 (Section[6](. Our results with dynamic 
workloads demonstrate that WFIT generates online index recom- 
mendations of high quality, even when compared to the best in- 
dices that could be chosen with advance knowledge of the complete 
workload. We also show that WFIT can benefit from good feedback 
in order to improve further the quality of its recommendations, but 
is also able to recover gracefully from bad advice. 

2 Preliminaries 

General Concepts. We model the workload of a database as a 
stream of queries and updates Q. We let g„ denote the n-th state- 
ment and Qjv denote the prefix of length TV. 

Define T as the set of secondary indices that may be created on 
the database schema. The physical database design comprises a 
subset of T that may change over time. Given a statement q £ Q 
and set of indices X C I, we use cost{q, X) to denote the cost 
of evaluating q assuming that X is the set of materialized indices. 
This function is possible to evaluate through the what-if interface 



of modern optimizers. Given disjoint sets X,Y C X, we define 
benefit^{Y, X) = cost{q,X) — cost{q,Y U X) as the differ- 
ence in query cost if Y is materialized in addition to X. Note that 
benefit ^(Y, X) may be negative, if q is an update statement and Y 
contains indices that need to be updated as a consequence of q. 

Another source of cost comes from adding and removing ma- 
terialized indices. We let S{X, Y) denote the cost to change the 
materialized set from X toY. This comprises the cost to create the 
indices inY — X and to drop the indices in X — Y. The 5 function 
satisfies the triangle inequality: S(X,Y) < 5(X,Z) + S{Z,Y). 
However, 5 is not a metric because indices are often far more ex- 
pensive to create than to drop, and hence symmetry does not hold: 
S{X, Y) / 5{Y, X) for some X, Y. 

Index Interactions. A key concern for index selection is the issue 
of index interactions. Two indices a and b interact if the benefit 
of a depends on the presence of b. As a typical example, a and 
b can interact if they are intersected in a physical plan, since the 
benefit of each index may be boosted by the other. Note, however, 
that indices can be used together in the same query plan without 
interacting. This scenario commonly occurs when indices are used 
to handle selection predicates on different tables. 

We employ a formal model of index interactions that is based 
on our previous work on this topic |16| . Due to the complexity 
of index interactions, the model restricts its scope to some subset 
J (Z X of interesting indices. (In our context, J is usually a set of 
indices that are relevant for the current workload.) The degree of 
interaction between a and b with respect to a query q is 

doiq{a, b) = max | benefit ^^{{a} , X) — benefit ^^{{a} , X U {b})\. 

It is straightforward to verify the symmetry doiq (a, b) = doiq (6, a) 
by expanding the expression of benefit^ in the metric definition. 
Overall, this degree of interaction captures the amount that the ben- 
efits of a and b affect each other. Given a workload Q, we say a, b 
interact if 3g G Q : doiq[a, b) > 0, and otherwise a, b are inde- 
pendent. 

Let {Pi, . . . , Pk} denote a partition of indices in J. Each Pk 
is referred to as a part. The partition is called stable if the cost 
function obeys the following identity for any X (Z J: 

cost(q,X) = cost(g, 0) -Ef=i benefit ^( X n Pk,<ll). (2.1) 

Essentially, a stable partition decomposes the benefit of a large set 
X into benefits of smaller sets XnPk. The upshot for index tuning 
is that indices can be selected independently within each Pk, since 
indices from different parts have independent benefits. As shown 
in 1 16], the stable partition with the smallest parts is given by the 
connected components of the binary relation {{a,b) j a,b interact}. 
The same study also provides an efficient algorithm to compute the 
binary relation and hence the minimum stable partition. 

In the worst case, the connected components can be quite large 
if there are many complex index interactions. In practice, the parts 
can be made smaller by ignoring weak interactions, i.e., index-pairs 
(a, b) where dotq{a, b) is small. Equation i2. 1 1 might not strictly 
hold in this case, but we can ensure that it provides a good approx- 
imation of the true query cost (that is still useful for index tuning) 
as long as the partition accounts for the most significant index in- 
teractions. We discuss this point in more detail in Section|5] 

3 Semi- Automatic Index Tuning 

At a high level, a semi-automatic tuning algorithm takes as input 
the current workload and feedback from the DBA, and computes 
a recommendation for the set of materialized indices. (Both in- 
puts are continuous and revealed one "element" at a time.) The 



DBA may inspect the recommendation at any time, and is solely 
responsible for scheduling changes to the materialized set. The on- 
line analysis allows the algorithm to adapt its recommendations to 
changes in the workload or in the DBA's preferences. Moreover, 
the feedback mechanism enables the DBA to pass to the algorithm 
domain knowledge that is difficult to obtain automatically. We de- 
velop formal definitions for these notions and for the overall prob- 
lem statement in the following subsection. 

We note that our focus is on the core problem of generating index 
recommendations, which forms the basic component of any index 
advisor tool. An index advisor typically includes other peripheral 
components, such as a user interface to visually inspect the current 
recommendation (9l|16| or methods to determine a materialization 
schedule for selected indices f 161. These components are mostly 
orthogonal to the index-recommendation component and hence we 
can reuse existing implementations. Developing components that 
are specialized for semi-automatic index tuning may be an interest- 
ing direction for future work. 

3.1 Problem Formulation 

Feedback Model. We use a simple and intuitive feedback model 
that allows the DBA to submit positive and negative votes accord- 
ing to current preferences. At a high level, a positive vote on index 
a implies that we should favor recommendations that contain a, un- 
til the workload provides sufficient evidence that a decreases per- 
formance. The converse interpretation is given for a negative vote 
on a. Our feedback model allows the DBA to cast several of these 
votes simultaneously. Formally speaking, the DBA expresses new 
preferences by providing two disjoint sets of indices , F~ C I, 
where indices in receive positive votes and indices in re- 
ceive negative votes. 

We say that the DBA provides explicit feedback when they di- 
rectly cast votes on indices. We also allow for implicit feedback 
that can be derived from the manual changes that the DBA makes 
to the index configuration. More concretely, we can infer a pos- 
itive vote when an index is created and a negative vote when an 
index is dropped. The use of implicit feedback realizes an unobtru- 
sive mechanism for automated tuning, where the tuning algorithm 
tailors its recommendations to the DBA's actions even if the DBA 
operates "out-of-band", i.e., without explicit communication with 
the tuning algorithm. 

Problem Formulation. A semi-automatic tuning algorithm re- 
ceives as input the workload stream Q and a stream V that rep- 
resents the feedback provided by the DBA. Stream V has elements 
of the form F = (F+,F") per our feedback model. Its contents 
are not synchronized with Q, since the DBA can provide arbitrary 
feedback at any point in time. We only assume that Q and V are 
ordered in time, and we may refer to Q U 1^ as a totally ordered 
sequence. The output of the algorithm is a stream of recommended 
index sets SCI, generated after each query or feedback element 
inQuV. We focus on online algorithms, and hence the computa- 
tion of 5* can use information solely from past queries and votes — 
the algorithm has absolutely no information about the future. 

In order to complete the problem statement, we must tie the al- 
gorithm's output to the feedback in V. Intuitively, we consider the 
DBA to be an expert and hence the algorithm should trust the pro- 
vided feedback. At the same time, the algorithm should be able 
to recover from feedback that is not useful for the subsequent state- 
ments in the workload. We bridge these somewhat conflicting goals 
by requiring each recommendation S to be consistent with recent 
feedback in V. To formally define consistency, let F^ be the set 
of indices which have received a vote after the most recent query. 



where the most recent vote was positive. Define F~ analogously 
for negative votes. The consistency constraint requires S to contain 
all indices in F+ and no indices in F~ , i.e., F^ C SASnF~ = 0. 

Consistency forces recommendations to agree with the DBA's 
cumulative feedback so long as the algorithm has not analyzed a 
new query in the input. This property is aligned with the assump- 
tion that the DBA is a trusted expert. Moreover, consistency en- 
ables an intuitive interface in the case of implicit feedback that 
is derived from the DBA's actions: without the consistency con- 
straint, it would be possible for the DBA to create an index a and 
immediately receive a recommendation to drop a (an inconsistent 
recommendation) even though the workload has not changed. 

At the same time, our definition implies that F^ = F^ — 
when a new query arrives. This says that votes can only force 
changes to the recommended configuration until the next query is 
processed, at which time the algorithm is given the option to over- 
ride the DBA's previous feedback. Of course, the algorithm needs 
to analyze the workload carefully before taking this option, and 
determine whether the recent queries provide enough evidence to 
override past feedback. Otherwise, it could appear to the DBA 
that the system is ignoring the feedback and changing its recom- 
mendation without proper justification. Too many changes to the 
recommendation can also hurt the theoretical performance of an 
algorithm, as we describe later. 

The Semi-Automatic Timing Problem: Given a workload Q 
and a feedback stream V of pairs {F~^ , F~), generate a recom- 
mended index set S Q X after each element in Q U V such that S 
obeys the consistency constraint. 

Note that user-specified storage constraints are not part of the 
problem statement. Although storage can be a concern in practice, 
the recommendation size is unconstrained because it is difficult to 
answer the question "How much disk space is enough?" before see- 
ing the size of recommended indices. Instead, we allow the DBA 
to control disk usage when selecting indices from the recommenda- 
tion!] ^° validate our choice, we conducted a small survey among 
DBAs of real- world installations. The DBAs were asked whether 
they would prefer to specify a space budget for materialized in- 
dices, or to hand-pick indices from a recommendation of arbitrary 
size. The answers were overwhelmingly in favor of the second op- 
tion. One characteristic response said "Prefer hand-pick from DBA 
perspective, as storage is not so expensive as compared to overall 
objective of building a highly scalable system." This does not im- 
ply we should recommend all possible indices. On the contrary, as 
we see below, the recommendation must account for the overhead 
of materializing and maintaining the indices it recommends. 

Performance Metrics. Intuitively, a good semi-automatic tun- 
ing algorithm should recommend indices that minimize the overall 
work done by the system, including the cost to process the work- 
load as well as the cost to implement changes to the materialized in- 
dices. The first component is typical for index tuning problems and 
it reflects the quality of the recommendations. The second compo- 
nent stems from the online nature of the problem: the recommen- 
dations apply to the running state of the system, and it is clearly 
desirable to change the materialized set at a low cost. Low mate- 
rialization cost is important even if new indices are built during a 
maintenance period, since these periods have limited duration and 
typically involve several other maintenance tasks (e.g., generation 
of usage reports, or backups). 

Formally, let A be a semi-automatic tuning algorithm, and define 

~i I — 1 1 — ■ 
Previous work |9, 16 1 and commercial systems provide tools to 

inspect index configurations, which may be adapted to our setting. 
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Figure 1: Components of the WFIT Algorithm. 



Sn as the recommendation that A generates after analyzing q„ and 
all feedback up to Qn+i- Also denote the initial set of indices as 
Sq. We define the following total work metric that captures the 
performance of A's recommendations: 

totWork{A,QN,V) = ^ cost(<3„, ^n) + 5(5'„_i, Sn) 



The value of totWork(A,QM,V) models the performance of a 
system where each recommendation S„ is adopted by the DBA 
for the processing of query g„. This convention follows common 
practice in the field of online algorithms 1 3 1 and is convenient for 
the theoretical analysis that we present later. In addition, this model 
captures the effect of the feedback in V, as each S„ is required to 
be consistent (see above). Overall, total work forms an intuitive 
objective function, as it captures the primary sources of cost, while 
incorporating the effect of feedback on the choices of the algorithm. 
The adoption of this metric does not change the application of semi- 
automatic tuning in practice: the tuning algorithm will still generate 
a recommendation after each element in Q U V, and the DBA will 
be responsible for any changes to the materialized set. 

It is clearly impossible for an online algorithm A to yield the 
optimal total work for all values of Qjv and V. Consequently, 
we adopt the common practice of competitive analysis: we mea- 
sure the effectiveness of A by comparing it against an idealized 
offline algorithm OPT that has advance knowledge of Qat and V 
and can thus generate optimal recommendations. Specifically, we 
say that A has competitive ratio c if totWork{A, Qn, V) < c ■ 
totWork{ov'Y , Qn, V) + q for any Qjv and V , where a is con- 
stant with respect to Qjv and V , and A and OPT choose recommen- 
dations from the same finite set of configurations. The competitive 
ratio c captures the performance of A compared to the optimal rec- 
ommendations in the worst case, i.e., under some adversarial input 
Qjv and V . In this work, we assume that V = for the purpose 
of competitive analysis, since V comes from a trusted expert and 
hence the notion of adversarial feedback is unclear in practice. Our 
theoretical results demonstrate that the derivation of c remains non 
trivial even under this assumption. Applying competitive analysis 
to the general case of V 7^ is a challenging problem that we leave 
for future work. 

3.2 Overview of Our Solution 

The remainder of the paper describes the WFIT algorithm for 
semi-automatic index tuning. Figure [T| illustrates WFIT's approach 
to generating recommendations based on the workload and DBA 
feedback. The approach starts with a candidate selection com- 
ponent, which generates indices that are relevant to the incoming 
queries. During candidate selection, WFIT also analyzes the inter- 
actions between candidate indices and uses these interactions to 



determine a stable partition of the candidates (see Section|2j. Then 
the output of candidate selection is a partitioned set of indices, as 
shown in Figure [T| Once these candidates are chosen, WFIT an- 
alyzes the benefit of the indices with respect to the workload in 
order to generate the final recommendation. The logic that WFIT 
uses to generate recommendations is based on the Work Function 
Algorithm (WFA) of Borodin and El-Yaniv |3J. The original ver- 
sion of WFA was proposed for metrical task systems (4| but we 
extend its functionality to apply to semi-automatic index selection. 
A separate instance of WFA analyzes each part of the candidate set 
and only recommends indices within that part. As we discuss later, 
this divide-and-conquer approach of WFIT improves the algorithm's 
performance and theoretical guarantees. Finally, the DBA may re- 
quest the current recommendation at any time and provide feedback 
to WFIT. The feedback is incorporated back into each instance of 
WFA and considered for the next recommendation. 

The following two sections present the full details of each com- 
ponent of WFIT shown in Figure [T| Section |4] defines WFA and 
describes how WFIT leverages the array of WFA instances for its 
recommendation logic. Section |5] completes the picture, with the 
additional mechanisms that WFIT uses to generate candidates and 
account for DBA feedback. 



4 A Work Function Algorithm 
for Index Tuning 

The index tuning problem closely follows the study of task sys- 
tems from online computation f4l. This allows us to base our rec- 
ommendation algorithm on existing principled approaches. In par- 
ticular, we apply the Work Function Algorithm |3 1 (WFA for short), 
which is a powerful approach to task systems with an optimal com- 
petitive ratio. 

In order to fit the assumptions of WFA, we do not consider the 
effect of feedback and we fix a set of candidate indices C C I from 
which all recommendations will be a drawn. In the next section, we 
will present the WFIT algorithm, which builds on WFA with support 
for feedback and automatic maintenance of candidate indices. 

4.1 Applying the Work Function Algorithm 

We introduce the approach of WFA with a conceptual tool that vi- 
sualizes the index tuning problem in the form of a graph. The graph 
has a source vertex 5*0 to represent the initial state of the system, as 
well as vertices X) for each statement g„ and possible index 
configuration X C C. The graph has an edge from 5*0 to (gi, X) 
for each X, and edges from (gn-i, X) to (qn, Y) for all X, Y and 
1 < n < N. The weight of an edge is given by the transition cost 
between the corresponding index sets. The nodes (q, X) are also 
annotated with a weight of cost{q, X). We call this the index tran- 
sition graph. The key property of the graph is that the totWork 
metric is equivalent to the sum of node and edge weights along the 
path that follows the recommendations. Figure|2]illustrates this cal- 
culation on a small sample graph. A previous study 1 2| has used this 
graph formulation for index tuning when the workload sequence is 
known a priori. Here, we are dealing with an online setting where 
the workload is observed one statement at a time. 

The internal state of WFA records information about shortest paths 
in the index transition graph, where the possible index configura- 
tions comprise the subsets of the candidate set C. More formally, 
after observing n workload statements, the inteiTial state of WFA 
tracks a value denoted Wn (S) for each index set S C C, as defined 



in the following recurrence: 

WniS) = mm{wn-i{X) + cost{q„,X) + 5{X,S)} (4.1) 



woiS) 



S{So,S) 



We henceforth refer to w„{S) as the work function value for S 
after n statements. As mentioned above, the work function can 
be interpreted in terms of paths in the index transition graph. In 
the case where n is positive, w„{S) represents the sum of (i) the 
cost of the shortest path from 5*0 to some graph node (q^, X), and 
(ii) the transition cost from X to S. The actual value of Wn{S) uses 
the X (Z C which minimizes this cost. We can think of wo{S) in a 
similar way, where the "path" is an empty path, starting and ending 
at So- Then the definition woiS) = S{So, S) has a natural analogy 
to the recursive case. 

Note that the total work of the theoretically optimal recommen- 
dations is equivalent to toiVForA:(Q,i, OPT, 0) = rnin5cc{f^n(5')}. 
Hence, the intuition is that WFA can generate good recommenda- 
tions online by maintaining information about the possible paths of 
optimal recommendations. 

Figure [3] shows the pseudocode for applying WFA to index tun- 
ing. All of the bookkeeping in WFA is based on the fixed set C of 
candidate indices. The algorithm records an array w that is indexed 
by the possible configurations (subsets of C). After analyzing the 
n-th statement of the workload, w[S] records the work function 
value 'Wn{S). The internal state also includes a variable currRec 
to record the current recommendation of the algorithm. 

The core of the algorithm is the analyzeQuery method. There 
are two stages to the method. The first stage updates the array w 
using the recurrence expression defined previously. The algorithm 
also creates an auxiliary array p. Each p[5'] contains index sets 
X such that a path from 5*0 to [qn,X) minimizes Wn{S). The 
second stage computes the next recommendation to be stored in 
currRec. WFA assigns a numerical score to each configuration S 
as score{S) — w[S] + S(S, currRec) and the next state must min- 
imize this score. To see the intuition of this criterion, consider a 
configuration X with a higher score than currRec, meaning that 
X cannot become the next recommendation. Then 

score{currRec) < score{X) 
=> Wn{currRec) — w„{X) < S{X, currRec). 

The left-hand side of the final inequality can be viewed as the ben- 
efit of choosing a new recommendation X over currRec in terms 
of the total work function, whereas the right side represents the 
cost for WFA to "change its mind" and transition from X back to 
currRec. When the benefit is less than the transition cost, WFA will 
not choose X over the current recommendation. This cost-benefit 
analysis helps WFA make robust decisions (see Theorem |4. l[ l. 

The recommendation S chosen by WFA must also appear in p[S] . 
Recall that p[5'] records states X s.t. there exists a path from So to 
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This small graph visualizes total work for a workload of three queries qi, q2, Q3, 
where recommendations are chosen between and {a}. The index a has cost 
20 to create and cost to drop. The highlighted path in the graph corresponds 
to an algorithm that recommends for qi and {a} for Q2, qs- The combined 
cost of edges and nodes in the path is 5(0, 0) + cost{qi, 0) + (5(0, {a}) + 
cost{q2, {a}) + S{{a}, {a}) + cost{q3, {a}) = 57. 

Figure 2: Index transition graph 



Data: Set C C I of candidate indices; Array w of work function values; 

Configuration currRec. 
Initialization: Candidates C and initial state So Q C given as input; 

w[5] = <5(5o, S) for each S C C; currRec = Sq. 

Procedure WFA. analyzeQuery (q) 

Input: The next statement q in the workload 

1 Initialize an'ays w' and p; 

2 foreach 5 C C do 

3 w'[5] = minxcc{w[X] -I- cost{q, X) + 5(X, 5)}; 

4 [ p[S] = {X <ZC\m'[S]= w[X] + cost{q, X) + 5(X, 5)}; 

5 Copy w' to w; 

6 foreach S C C do score{S) 4- wf^] + S{S, currRec); 

7 currRec <— arg minggp[s] {score(S)}; 

Function W FA. recommend {) 
1 return currRec; 



Figure 3: Pseudocode for WFA. 



{qn,X) that minimizes Wn{S). The condition specifies that X = 
S for one such path, and hence w„{S) — Wn~i{S) + cost{q, S). 
An important result from Borodin et al. (|3|, Lemma 9.2) shows 
that this condition is always satisfied by a state with minimum 
score. In other words, the criterion S G p[S] is merely a tie-breaker 
for recommendations with the minimum score, to favor configura- 
tions whose work function does not include a transition after the 
last query is processed. This is crucial for the theoretical guaran- 
tees of WFA that we discuss later. 

Example 4.1. The basic approach of WFA can be illustrated 
using the scenario in Figure [2] The actual recommendations of 
WFA will be the same as the highlighted nodes. Before the first 
query is seen, the work function values are initialized as 

wo{9) = 0, too({a}) = 20 

based on the transition cost from the initial configuration So = 0. 
After the first query, the work function is updated using \A.i\ : 

«;i(0) = 15, uii({a}) = 25. 

These values are based on the paths 0^0 and {a} respec- 
tively^ The scores are the same as the respective work function 
values (5(0, 0) — S{{a}, 0) = a/ line^of^Vh. analyzeQuery), 
hence remains as WFA 's recommendation due to its lower score. 
After q2, the work function values are both 

W2(0) = W2{{a}) = 27. 

Both values use the path ^ {a} -» {a}. The calculation of W2 (0) 
also includes the transition S{{a},0), which has zero cost The cor- 
responding scores are again equal to the work function, but here the 
tie-breaker comes into play: {a} is preferred because it is used to 
evaluate q2 in both paths, hence WFA switches its recommendation 
to {a}. Finally, after qz, the work function values are 



■ 47. 



«>3(0) = 42, u>3({a}) 

based on paths -» {a} {a} and -» {a} {a} {a} 
respectively. The actual scores must also account for the current 
recommendation {a}. Following line^of WFA. analyzeQuery, 

score{$) = 62, score{{a}) = 47. 

The recommendation of WFA remains {a}, since it has a lower 
score. This last query illustrates an interesting property of WFA: 



For example 4. 1 we abuse notation and use index sets X in place 



of the graph nodes {q„,X) 



although the most recent query has favored dropping a, the recom- 
mendation does not change because the difference in work function 
values is too small to outweigh the cost to materialize a again. | 

As a side note, observe that the computation of Wn{S) requires 
computing cost{q, X) for multiple configurations X. This is feasi- 
ble using the what-if optimizer of the database system. Moreover, 
recent studies 1 12 , 8 1 have proposed techniques to speed up succes- 
sive what-if optimizations of a query. These techniques can readily 
be applied to make the computation of w„ very efficient. 

WFA's Advantage: Competitive Analysis, wfa is a seemingly 
simple algorithm, but its key advantage is that we can prove strong 
guarantees on the performance of its recommendations. 

Borodin and El-Yaniv [3] showed that WFA has a competitive 
ratio of 2(T — 1 for any metrical task system with a possible config- 
urations, meaning that its worst-case performance can be bounded. 
Moreover, WFA is an optimal online algorithm, as this is the best 
competitive ratio that can be achieved. These are very powerful 
properties that we would like to transfer to the problem of index 
recommendations. However, the original analysis does not apply 
in our setting, since it requires 5 to be a metric, and our definition 
of S is not symmetric. One of the technical contributions of this 
paper is to show how to overcome the fact that 5 is not a metric, 
and extend the analysis to the problem of index recommendations. 

Theorem 4.1. The wfa algorithm, as shown in Figure^ has 
a competitive ratio 0/2''''^^ — 1. (Proof in the appendix) 

This theoretical guarantee bolsters our use of WFA to generate 
recommendations. The competitive ratio ensures that the recom- 
mendations do not have an arbitrary effect on performance in the 
worst case. We show empirically in Section[6]that the average-case 
performance of the recommendations can be close to optimal. This 
behavior is appealing to DBAs, since they would not want to make 
changes that can have unpredictably bad performance. 

4.2 Partitioning the Candidates 

In the study of general task systems, the competitive ratio of WFA 
is theoretically optimal |4|. However, the algorithm has some draw- 
backs for the index recommendation problem, since it becomes in- 
feasible to maintain statistics for every subset of candidates in C as 
the size of C increases. The competitive ratio 2''''"'"^ — 1 also be- 
comes nearly meaningless for moderately large sets C. Motivated 
by these observations, we present an enhanced algorithm WFA+, 
which exploits knowledge of index interactions to reduce the com- 
putational complexity of WFA, while enabling stronger theoretical 
guarantees. 

The strategy of WFA+ employs a stable partition {Ci, . . . , Ck} 
of C, as defined in Section|2] The stable partition guarantees that in- 
dices in Ck do not interact with indices in any other part Ci ^ Ck- 
This is formalized by l |2.1^ , which shows that each part d makes 
an independent contribution to the benefit. Moreover, it is straight- 
forward to show that 5[X, Y) = J]^. 5[X r\Ck,Y r\ Ck), i.e., we 
can localize the transition cost within each subset Ck- These obser- 
vations allow WFA+ to decompose the objective function totWork 
into K components, one for each Ck, and then select indices within 
each subset using separate instances of WFA. 

We define WFA^ as follows. The algorithm is initialized with 
a stable partition {Ci, . . . , Ck} of C, and initial configuration 5*0. 
For fc = 1,. . . ,K, WFA+ maintains a separate instance of WFA, de- 
noted WFA^*'. We initialize WFa'*' with candidates Ck and initial 
configuration So n Cfc. The interface of WFA+ follows WFA: 

• WFA^ .analyzeQuery{q) calls WFA^''\analyzeQuery{q) for each 
k^l,---,K- 



• WFA^. recommenrf() returns IJj, wfa' ' .recommend {). 

On the surface, WFA^ is merely a wrapper around multiple in- 
stances of WFA, but the partitioned approach of WFA+ provides sev- 
eral concrete advantages. The division of indices into a stable par- 
tition implies that WFA^ must maintain statistics on only X^fc 2''^'° 
configurations, compared to the 2'''' states that would be required 
to monitor all the indices in WFA. This can simplify the book- 
keeping massively: a back-of-the-envelope calculation shows that 
if WFA+ is given 32 indices partitioned into subsets of size 4, then 
only 128 configurations need to be tracked, whereas WFA would re- 
quire more than four billion states. We prove that this simplification 
is lossless, i.e., that WFA"*" selects the same indices as WFA. 

Theorem 4.2. If {Ci, ... ,Ck} is a stable partition of C, then 
WFA^ on {Ci, . . . , Ck} will make the same recommendations as 
WFA on C. (Proof in the appendix) 

It immediately follows that WFA"*" inherits the competitive ratio 
of WFA. However, the power of WFA+ is that it enables a much 
smaller competitive ratio by taking advantage of the stable parti- 
tion. 

Theorem 4.3. WFA+ has a competitive ratio o/2'''"="'+^ — 1, 
where Cmax ~ maxfc{|Cfc|}. (Proof in the appendix) 

Hence the divide-and-conquer strategy of WFA+ is a win-win, as 
it improves the computational complexity of WFA as well as the 
guarantees on performance. Observe that WFA^ matches the com- 
petitive ratio of 3 that the online tuning algorithm of Bruno and 
Chaudhuri |5| achieves for the special case \C\ — 1 (the compet- 
itive analysis in (5] does not extend to a more general case). The 
competitive ratio is also superior to the ratio > 8(2'''' — 1) for the 
OnlinePD algorithm of Malik et al. |I0| for a related problem in 
online tuning. 

5 The WFIT Algorithm 

We introduced WFA+ in the previous section, as a solution to the 
index recommendation problem with strong theoretical guarantees. 
The two limitations of WFA^ are (i) it does not accept feedback, 
and (ii) it requires a fixed set of candidate indices and stable parti- 
tion. In this section, we define the WFIT algorithm, which extends 
WFA^ with mechanisms to incorporate feedback and automatically 
maintain the candidate indices. 

Figure |4] shows the interface of WFIT in pseudocode. The meth- 
ods analyzeQuery and recommend perform the same steps as the 
corresponding methods of WFA+. In analyzeQuery, WFIT takes 
additional steps to maintain the stable partition {Ci , . . . , Ck } ■ This 
work is handled by two auxiliary methods: chooseCands deter- 
mines what the next partition should be, and repartition reorga- 
nizes the data structures of WFIT for the new partition. Finally, 
WFIT adds a new method feedback, which incorporates explicit or 
implicit feedback from the DBA. 

In the next subsection, we discuss the feedback method. We then 
provide the details of the chooseCands and repartition methods 
used by analyzeQuery. 

5.1 Incorporating Feedback 

As discussed in Section[3] the DBA provides feedback by casting 
positive votes for indices in some set and negative votes for a 
disjoint set . The votes may be cast at any point in time, and the 
sets may involve any index in C (even indices that are not 

part of the current recommendation). This mechanism is captured 
by a new method feedback{F^ , F~ ). The DBA can call feedback 
explicitly to express preferences about the index configuration, and 



we also use feedback to account for the implicit feedback from 
manual changes to the index configuration. 

Recall from Section[3]that the recommendations must be consis- 
tent with recent feedback, but should also be able to recover from 
poor feedback. Our approach to guaranteeing consistency is sim- 
ple: Assuming that currRec is the current recommendation, the 
new recommendation becomes currRec — F~ U . Since WFIT 
forms its recommendation as currRec^, where currRec^ is the 
recommendation from WFA running on part Ck, we need to modify 
each currReCk accordingly. Concretely, the new recommendation 
for Ck becomes currReck — F~ U {F^ n Ck)- 

The recoverability property is trickier to implement properly. 
Our solution is to adjust the scores in order to appear as if the work- 
load (rather than the feedback) had led WFIT to recommend creat- 
ing F'^ and dropping F^ . With this approach, WFIT can naturally 
recover from bad feedback if the future workload favors a differ- 
ent configuration. To enforce the property in a principled manner, 
we need to characterize the internal state of each instance of WFA 
after it generates a recommendation. Recall that WFA selects its 
next recommendation as the configuration that minimizes the score 
function. Let us assume that the selected configuration is Y, which 
differs from the previous configuration by adding indices and 
dropping indices Y^ . If we recompute score after Y becomes the 
current recommendation, then we can assert the following bound 
for each configuration 5": 

score{S) — score{Y) > 

5{S,S -Y- UY+) + S(S -Y- UY+,3) (5.1) 



Essentially, this quantity represents the minimum threshold that 
score{S) must overcome in order to replace the recommendation 
Y. Hence, in order for the internal state of WFa'*^' to be consistent 
with switching to the new recommendation currReck, we must en- 
sure that score{S) — score{currReck), or the equivalent ex pres 
sion w'^^'fS] + S{S, currReck) — w^''^[currReck], respects i5.1 
This can be achieved by increasing w'''' [S] accordingly. 



Figure|4]shows the pseudocode for feedback based on the previ- 
ous discussion. For each part Ck of the stable partition, feedback 
first switches the current recommendation to be consistent with the 
feedback (line[4]l. Subsequently, it adjusts the value of w''''[S] for 
each S C Cfe to enforce the bound \5.l\ on score{S). 

5.2 Maintaining Candidates Automatically 

The analyzeQuery method of WFIT extends the approach of 
WFA+ to automatically change the stable partition as appropriate 
for the current workload. We present these extensions in the re- 
mainder of this section. We first discuss the repartition method, 
which updates WFIT's internal state according to a new stable par- 
tition. Finally, we present chooseCands, which determines what 
that stable partition should be. 

5.2.1 Handling Changes to the Partition 

Suppose that the repartition method is given a stable partition 
{Di, . . . , Dm} for WFIT to adopt for the next queries. We require 
each of the indices materialized by WFA to appear in one of the sets 
D\, . . . , Dm, in order to avoid inconsistencies between the inter- 
nal state of WFIT and the physical configuration. In this discussion, 
we do not make assumptions about how {Di , . . . , Dm} is chosen. 
Later in this section, we describe how chooseCands automatically 
chooses the stable partition that is given to repartition. 

Unmodifled Candidate Set. We initially consider the case where 
the new partition is over the same set of candidate indices, i.e.. 



Data: Current set C of candidate indices; 
Stable partition {Ci , . . . , Ck} of C; 
WFA instances WFA^^), . . . , WFa'^); 
Initialization: Initial index set So is provided as input; 

C = So,K = \So\ and d = {aj where 1 < « < |So| and 
ai, . . . , QiSqI are the indexes in So ; 
for fc 1 to it do 

WFa(*) 4- instance of WFA with candidates Ck 
and initial coniiguration Ck H So 

Procedure wfit . analyzeQuery (q) 
Input: The next statement q in the workload. 

{Di, . . . , Dm} chooseCands (q) ; // Figure[6\ 

it {Di,..., Dm} 7^{Ci,...,Ck} then 

/ / Replace {Ci,...,Ck} with {Di , Dm}- 
repartition{{Di, . . . , Dm}) : / / figurepl 

for fc <— 1 to ii" do WFA^''\analyzeQuery{q); 



Function WFIT. recommend () 
1 return Uj. WFa('°). recommend(); 

Procedure WFlT./ee(ibacfc(_F+ ,F~) 

Input: Index sets _F+, F~ C C with positive/negative votes. 

1 for A; -f— 1 to K do 

2 Let w'''' denote the work function of WFA^^' ; 

3 Let currReCk denote the cun'ent recommendation of WFa''"'; 

4 currReCk <— currReCk — F~ U (F+ H Ck); 

5 for 5 C Ck do 

6 S<^°°=<(-S'-F-U(F+nCfc); 

7 mm«if •!- (5(S',S'=°"=) + 5(5'=°"=,5); 

8 diff ^ wC") [S] + (5(5, currReCk) - wC-') [currReCk]; 

9 if diff < minDiff then 

10 |_ Increase w'''' [S] by minDiff — diff; 



Figure 4: Interface of wfit. 

[Jk=i Cfc ^ Um=i -^"1- The original internal state of WFIT corre- 
sponds to a copy of WFA for each stable subset Ck- The new parti- 
tion requires a new copy of WFA to be initialized for each new stable 
subset Dm - The challenge is to initialize the work function values 
corresponding to Dm in a meaningful way. We develop a gen- 
eral initialization method that maintains an equivalence between 
the work function values of {Di, . . . , Dm} and {Ci, . . . , Ck}, 
assuming that both partitions are stable. 

We describe the reinitialization of the work function with an ex- 
ample. Assume the old stable partition is Ci — {a},C2 ~ {&}, 
and the new stable partition has a single member Di — {a, b}- Let 
w'^' , w*^^' be the work function values maintained by WFIT for the 
subsets Ci , C2 . Let Wn be the work function that considers paths in 
the index transition graph with both indices a, b, which represents 
the information that would be maintained if a, b were in the same 
stable subset. In order to initialize work function values for Di, 
we observe that the following identity follows from the assumption 
that {Ci, C2} is a stable partition: 

Wn{S) = w^^\S n {a}) + w^^\S n {b}) ^ cost{q„il)) 



This is a special case of Lemma |B.l[ which we prove in Appendix|B] 
The bottom line is that it is possible to reconstruct the values of the 
work function to„ using the work functions within the smaller par- 
titions. For the purpose of initializing the state of WFA, the final 
sum may be ignored: the omission of this sum increases the scores 
of each state S by the same value, which does not affect the deci- 



Procedure repartition{{Di , . . . , -Dm}) 
Input: The new stable partition. 

/ / Note: Di, . . . , Dm must cover materialized indices 

1 Let w(*' denote the work function of WFA^*' ; 

2 Let currRec denote the current recommendation of WFIT; 

3 for m <— 1 to M do 

4 Initialize array x'™) and configuration variable newReCm', 

5 foreach X e 2^™ do 

6 x(-)[x]^ELi«'('=nCfcnx]; 

7 [_ x(™) [X] ^ x(™) [X] + S{So nDm-C,X -C); 

8 newRec„i <— Dm, n currRec; 

9 Set {-Di, . . . , -Da/} as the stable partition, where Dm is tracked by a 
new instance WFA^™' with work function x^'"' and state newReCm', 

Figure 5: The repartition method of WFIT. 

sions of WFA. Based on this reasoning, our repartitioning algorithm 
would initialize Di using the array x defined as follows: 

x[0] w(i) [0] + [0] x[{a}] ^ w(i) [{a}] + w(2) [0] 

x[{fe}] ^ w(l) [0] + w(2) [{6}] x[{a, b}] ^ y/W [{„}] + w^^) [{b}] 

We use an analogous strategy to initialize the work function when 
repartitioning from Di to Ci , C2: 

w(i) [0] <- x[0] w(2)[0]^x[0] 
wW[{a}]4-x[{a}] w(2)[{6}]^x[{6}] 

Again, note that these assignments result in work function values 
that would be different if Ci , C2 were used as the stable partition 
for the entire workload. The crucial point is that each work function 
value is distorted by the same quantity (the omitted sum), so the 
difference between the scores of any two states is preserved. 

The pseudocode for repartition is shown in Figure |5] For each 
new stable subset Dm, the goal is to initialize a copy of WFA with 
candidates Dm- The copy is associated with an array x'™' that 
stores the work function values for the configurations in 2^'" . For 
a state X C Dm, the value x'"''[X] is initialized as the sum of 
w^'^'fX n Ck], i.e., the work function values of the configurations 
in the original partition that are maximal subsets of X (line[6](. This 
initialization follows the intuition of the example that we described 
previously, since the stable partition {Ci, . . . , Ck} implies that 
X n Cfc is independent from X f] Ci for k ^ I. Line |7] makes a 
final adjustment for new indices in X, but this is irrelevant if the 
candidate set does not change (we will explain this step shortly). 
Finally, the current state corresponding to Dm. is initialized by tak- 
ing the intersection of currRec with Dm- 

Overall, repartition is designed in order for the updated inter- 
nal state to select the same indices as the original state, provided 
that both partitions are stable. This property was illustrated in the 
example shown earlier. It is also an intuitive property, as two stable 
partitions record a subset of the same independencies, and hence 
both allow WFIT to track accurate benefits of different configura- 
tions. A more formal analysis of repartition would be worthwhile 
to explore in future work. 

Modified Candidate Set. We now extend our discussion to the 
case where the new partition is over a different set of candidate 
indices, i.e., Ul^i 7^ Um=i repartition method 

(Figure |5j can handle this case without modifications. The only 
difference is that line |7] becomes relevant, and it may increase the 
work function value of certain configurations. It is instructive to 
consider the computation of x*™' [X] when X contains an index a 
which did not previously appear in any C'k or the initial state 5*0. 



Data: Index set W 3 C from which to choose candidate indices; 
Array idxStats of benefit statistics for indices in U\ 
Array intStats of interaction statistics for pairs of indices in TJ- 

Procedure chooseCands(q) 

Input: The next statement q in the workload. 

Output: Di , . - - , Dm, a new partitioned set of candidate indices. 

Knobs: Upper bound idxCnt on number of indices in output; 

Upper bound stateCnt on number of states X]m 2l^'" ' . 

Upper bound histSize on number of queries to track in statistics 

1 W W U extractlndices{q); 

2 IBGq ^ computeIBG{q); 1 1 Based on |76^ 

3 updateStats{IBGq); 

4 JVl {a £ C I a is materialized}; 

5 C <- U topIndices{U - M, idxCnt - \M\); 

6 {Di, . . . , Dm} choosePartition{T> , stateCnt); 

7 return {-Di, . . . , Dm}; 

Figure 6: The chooseCands Method of wfit. 

Since a is a new index, it does not belong to any of the original 
subsets Ck , and hence the cost to materialize a will not be reflected 
in the sum Y.k ^ ^k]- Since x'™' [X] includes a transition 

to an index set with a materialized, we must add the cost to mate- 
rialize a as a separate step. This idea is generalized by adding the 
transition cost on line|7] The expression is a bit complex, but we 
can explain it in an alternative form 5{Sof^Dm —C, X n Dm —C), 
which is equivalent because X C Dm- In this form, we can make 
an analogy to the initialization used for the work function before 
the first query, for which we use wo{X) = S{So,X). The ex- 
pression used in line|7]computes the same quantity restricted to the 
indices {Dm — C) that are new within Dm- 

5.2.2 Choosing a New Partition 

As the final piece of WFIT, we present the method chooseCands, 
which automatically decides the set of candidate indices C to be 
monitored by WFA, as well as the partition {Ci , . . . , Ck} of C- 

At a high level, our implementation of chooseCands analyzes 
the workload one statement at a time, identifying interesting in- 
dices and computing statistics on benefit interactions. These statis- 
tics are subsequently used to compute a new stable partition, which 
may reflect the addition or removal of candidate indices or changes 
in the interactions among indices. As we will see shortly, several 
of these steps rely on simple, yet intuitive heuristics that we have 
found to work well in practice. Certainly, other implementations of 
chooseCands are possible, and can be plugged in with the remain- 
ing components of WFIT. 

The chooseCands method exposes three configuration variables 
that may be used to regulate its analysis. Variable idxCnt speci- 
fies an upper bound on the number of indices that are monitored 
by an instance of WFA, i.e., idxCnt > \C\ = \Ck\- Variable 
stateCnt specifies an upper bound on the number of configura- 
tions tracked by WFIT, i.e., stateCnt > J2k S'"^*"'. If the mini- 
mal stable partition does not satisfy these bounds, chooseCands 
will ignore some candidate indices or some interactions between 
indices, which in turn affects the accuracy of WFIT's internal statis- 
tics. Variable histSize controls the size of the statistics recorded 
for past queries. Any of these variables may be set to 00 in order 
to make the statistics as exhaustive as possible, but this may result 
in high computational overhead. Overall, these variables allow a 
trade-off between the overhead of workload analysis and the effec- 
tiveness of the selected indices. 

Figure|6]shows the pseudocode of chooseCands. The algorithm 
maintains a large set of indices U, which grows as more queries 
are seen. The goal of chooseCands is to select a stable partition 



over some subset T> C W. To help choose the stable partition, the 
algorithm also maintains statistics for U in two arrays: idxStats 
stores benefit information for individual indices and intStats stores 
information about interactions between pairs of indices within U. 

Given a new statement q in the workload, the algorithm first 
augments U with interesting indices identified by extractlndices 
(line[T]l. This function may be already provided by the database 
system (e.g., as with IBM DB2), or it can be implemented ex- 
ternally fP, l5l. Next, the algorithm computes the index benefit 
graph [16] (IBG for short) of the query (line|2](. The IBG com- 
pactly encodes the costs of optimized query plans for all relevant 
subsets of hi. As we discuss later, updateStats uses the IBG to 
efficiently update the benefit and interaction statistics (line[3](. The 
next step of the algorithm determines the new set of candidate in- 
dices V that should be monitored by WFIT for the upcoming work- 
load, with an invocation of toplndices on line[5] We ensure that D 
includes the currently materialized indices (denoted A4), in order 
to avoid overriding the materializations chosen by WFA. Finally, 
chooseCands invokes choosePartiUon to determine the partition 
Di, . . . , Dm of O, and returns the result. 

To complete the picture, we must describe the methodology that 
toplndices and choosePartiUon use to decide the new partition 
of indices, and the specific bookkeeping that updateStats does to 
enable this decision. 

The topIndices{X,u) Method. The job of toplndices {X,u) is 
to choose at most it candidate indices from the set X that have the 
highest potential benefit. 

We first describe the statistics used to evaluate the potential bene- 
fit of a candidate index. For each index a, the idxStats array stores 
entries of the form (n, /3„), where n is a position in the workload 
and Pn is the maximum benefit of a for query g„. The maximum 
benefit is computed as /3„ = maxxcw benefit ^^{{a} , X) . The 
cell idxStats [o] records the histSize most recent entries such that 
Pn > 0. These statistics are updated when chooseCands invokes 
updateStats on line[3] The function considers every index a that is 
relevant to q, and employs the IBG of query q in order to compute 
/3n efficiently. If > then {n,l3n) is appended to tdxStats[a] 
and the oldest entry is possibly expired in order to keep histSize 
entries in total. 

Based on these statistics, toplndices {X, u) returns a subset Y C 
X with size at most u, which becomes the new set of indices mon- 
itored by WFIT. The first step of toplndices computes a "cur- 
rent benefit" for each index in X, which captures the benefit of 
the index for recent queries. We use benefit*pf{a) to denote the 
current benefit of a after observing A'' workload statements, and 
compute this value as follows. If idxStats[a] = after N state- 
ments, then benefi,t*j^f{a) is zero. Otherwise, let idxStats[a] — 
(ni, 6i), . . . , (uLjbL) such that ni > ■ ■ • > hl- Then 

K / \ ^1 ^ 

benefit,^ [a) = max . 

^ i<i<L N -ne + 1 

For each £ — 1, . . . ,L, this expression computes an average benefit 
over the most recent A'^ — + 1 queries, and we take the maxi- 
mum over all £. Note that a large value of ne results in a small 
denominator, which gives an advantage to indices with recent ben- 
efit. This approach is inspired by the LRU-K replacement policy 
for disk buffering [ 1 1 J. 

The second step of topIndices(X, u) uses the current benefit to 
compute a score for each index in X, and returns the u indices with 
the highest scores. If a G A n C (i.e., a is currently monitored by 
WFA), the score of a is simply benefit* [a). The score of other 
indices b £ A — C is benefit* (b) minus the cost to materialize b. 



This means that b requires extra evidence to evict an index in C, 
which helps C be more stable. 

The choosePartition{T> , stateCnt) method. Conceptually, the 
stable partition models the strongest index interactions for recent 
queries. We first describe the statistics used to estimate the strength 
of interactions, and then the selection of the partition. 

The statistics for choosePartition are based on the degree of in- 
teraction doiq{a, b) between indices a,b £ U for a workload state- 
ment q (Section|2]l. Specifically, we maintain an array intStats that 
is updated in the call to updateStats (which also updates idxStats 
as described earlier). The idea is to iterate over every pair (a, b) 
of indices in the IBG, and use the technique of f 16] to compute 
d = doiq^{a,b). The pair (n, d) is added to mtStats[a,b] if 
d > 0, and only the histSize most recent pairs are retained. 

We use intStats[a, b] to compute a "current degree of interac- 
tion" for a, b after A^ observed workload statements, denoted as 
doi*pf(a, b), which is similar to the "current benefit" described ear- 
lier. If intStats[a, 6] = then we set doi*i^{a, b) — 0. Otherwise, 
let intStats[a,b] = (ni, di), . . . , (nj,, dj^) for ni > ■■■ > n^, 
and 

di H + de 



doi*f^{a, b) = 



l<t<L N 



+ 1 



To compute the stable partition, we conceptually build a graph 
where vertices correspond to indices and edges correspond to pairs 
of interacting indices. Then a stable partition is a clustering of the 
nodes so that no edges exist between clusters. In the context of 
chooseCands, we are interested in partitions {Pi, . . . , Pm} such 
that X^m 2'^™' — stateCnt. Since there may exist no stable par- 
tition that obeys this bound, our approach is to ignore interactions 
until a feasible partition is possible. This corresponds to dropping 
edges from the conceptual graph, until the connected components 
yield a suitable clustering of the nodes. 

An important question is which interactions to ignore. Our strat- 
egy is to minimize the error that the partition introduces in the for- 
mula for query cost ( |2.1[ l, which is the basis of all statistics tracked 
by WFIT. It is straightforward to show that the error in l |2.1^ is 
bounded by the sum of doi values for ignored interactions. Hence 
we define the loss of a partition P — {Pi , . . . , Pm} as 

loss(P) = X] 5Z 5Z d.oi*t^{a,b). 

i<j aePi bePj 

In the graph-based interpretation, this corresponds to the sum of 
edge weights for edges that cross clusters. 

Figure |7] shows the pseudocode for function choosePartition 
that computes the new stable partition. The goal is to return a 
feasible partition that minimizes loss. We employ a randomized 
approach that finds several feasible partitions and returns the one 
with the least loss. As a baseline solution, the function consid- 
ers the existing stable partition, augmented with singleton parts for 
the new indices in D. It then performs RAND_CNT random- 
ized iterations, where RAND_CNT is a parameter of the algo- 
rithm. Each iteration has two stages. The first stage simply merges 
singleton sets that exhibit a high degree of interaction. The pair 
({a}, {6}) to merge is chosen randomly with weight proportional 
to doj^(a, 6). The second stage is similar, with a different weight- 
ing scheme. Given two sets A and B, it assigns a weight propor- 
tional to Ei,6B dot*M{a, 6)/(2l^l + l^l - 2l^l - 2l^l). The 

normalization accounts for the additional number of configurations 
that will result from merging A and B. Essentially, the weight 
assigned to {A,B) represents the increase in loss{P) per addi- 
tional state tracked by WFIT, and hence the merging favors small 



Function choosePartition(T>, stateCnt) 
Input: Indices D to partition; 

Bound stateCnt > J2 2l^'"l for the output {Pi, . . . , Pm] 

1 bests oln <— 0; bestLoss <— oo; 

/ / Try a baseline partition that is similar to the current one 

2 Let {Ci , . . . , Ck } denote the current partition and C = IJ^ Cfc ; 

3 Initialize {C[ , . . . , C^,} by removing C -V from {Ci , . . . , Ca'}; 

4 P^{C(,...,C^.}; 

5 foreach a&V-CAo Add {a} to P; 

6 if P is feasible (i.e., satisfies the bound stateCnt) then 

7 1^ bestSoln 4— P; bestLoss = loss{P) ; 

/ / Try additional random partitions 

8 for i <- 1 to RAND_CNT do 
P <— a partition of 15 in singletons; 
while true do 

Let {Pi , . . . , P]\j } denote the contents of P; 
E -f- {{Pi. , Pj} \loss ({Pi , Pj}) > A feasible to merge Pi , Pj} ; 
if S = then break; 

else if El = {{P^,Pj} eE \ 1 = |Pi| = |Pj|} ^ 0then 
Choose random {Pi, Pj} £ Ei with probability 



9 
10 
11 
12 
13 
14 
15 

16 
17 

18 

19 
20 



proportional to loss({Pi, Pj}) : 



else 



Choose {Pi,Pj} G E with probability proportional to 

loss({P„Pj})/{2\P^\ + \P^\ -2l^'l -2l^^l); 
P <— result of merging Pi, Pj in P ; 



if /oss(P) < bestLoss then 
bestSoln <— P; bestLoss ■ 



loss{P) : 



21 return bestSoln; 



Figure 7: Function choosePartition. 



sets whose indices have strong interactions. The function returns 
the best partition found across all iterations. 

This concludes the final piece of the WFIT algorithm. As a final 
note, observe that the methods of WFIT use strategies that are quite 
orthogonal. In particular, the method repartition does not depend 
on the specific heuristics that chooseCands uses to determine the 
candidate indices. With this design, it is straightforward to substi- 
tute chooseCands with alternate strategies for candidate selection 
and partitioning. There is a broad design space for this component 
of WFIT, and this would be an interesting direction for future work. 

6 Experimental Study 

In this section, we present an empirical evaluation of WFIT us- 
ing a prototype implementation that works as middleware on top 
of an existing DBMS. The prototype, written in Java, intercepts the 
SQL queries and analyzes them to generate index recommenda- 
tions. The prototype requires two services from the DBMS: access 
to the what-if optimizer, and an implementation of the 
extractlndices(q) method (line[T]in Figure|6|. This design makes 
the prototype easily portable, as these services are common primi- 
tives found in index advisors |17|[T1. 

We conducted experiments using a port of the prototype to the 
IBM DB2 Express-C DBMS. The port uses DB2's design advi- 
sor 1 17 1 to provide what-if optimization and extractlndices{q). 
Unless otherwise noted, we set the parameters of WFIT as follows: 
idxCnt = 40, stateCnt — 500, and histSize = 100. All exper- 
iments were run on a machine with two dual-core 2GHz Opteron 
processors and 8GB of RAM. 



6.1 Methodology 



adaptatiorj^of the state-of-the-art online tuning algorithm of Bruno 
and Chaudhuri |5|. BC analyzes the workload using ideas simi- 
lar to WFIT, except that it always employs a stable partition corre- 
sponding to full index independence, i.e., each part contains a sin- 
gle index. After a query is analyzed, BC heuristically adjusts the 
measured index benefits to account for specific types of index in- 
teractions. The principled handling of index interactions is a major 
difference between WFIT and BC. 

The second alternative is OPT, which has full knowledge of the 
workload and generates the optimal recommendations that mini- 
mize total work. OPT provides a baseline for the best-case perfor- 
mance of any online index recommendation algorithm. 

In order to make a meaningful comparison between these algo- 
rithms, some of our experiments use a fixed set of candidates C and 
stable partition {Ci, . . . , Ck} throughout the workload. In this 
way, the algorithms select their recommendations from the same 
configuration space, and our experiments focus on the recommen- 
dation logic. This approach requires a simplification of WFIT so 
that chooseCands always returns {Ci, . . . , Ck}- Our final exper- 
iment compares the simplified version of WFIT to the full version 
that allows chooseCands to modify the stable partition throughout 
the workload. 

Data Sets and Workloads. We base the experimental study on an 
index tuning benchmark that we introduced in our previous work ||15J . 
The benchmark is designed to stress test the effectiveness of online 
tuning algorithms, and it has already been used to compare exist- 
ing methods. The benchmark simulates a system hosting multiple 
databases using the synthetic data sets TPC-C, TPC-H and TPC-E 
and the real-life data set NREF, with a total of 2.9GB of base-table 
data. We note that the database size is not a crucial statistic for our 
study, as we evaluate the performance of index-tuning algorithms 
using the optimizer's cost model (see discussion below). 

We use the complex workload defined by the benchmark, which 
includes SQL query and update statements. Each statement in- 
volves a varying number of joins and selection predicates of mixed 
selectivity. The following is an example query from the workload: 

SELECT count (*) 

FROM tpce . security tablel, tpce . company table2, 

tpce . dailyjnarket tableO 
WHERE tablel. s_pe BETWEEN 63.278 AND 86.091 

AND tablel . s_exch_date BETWEEN '1995-05-12-01.46.40' 

AND '2006-07-10-01.46.40' 
AND table2 . co_open_date BETWEEN '1812-08-05-03.21.02' 

AND '1812-12-12-03.21.02' 
AND tablel . s_symb - tableO . din_s_symb 
AND table2 . co_id=tablel . s_co_id 

And the following is an example update: 

UPDATE tpch . lineitem 

SET l_tax = l_tax + RANDOM.SIGN *0 . 000001 

WHERE l_extendedprlce BETWEEN 65522.378 AND 66256.943 

This update statement uses a user-defined function rand om_s i GN ( ) 
which randomly returns 1 or —1 with equal probability. 

The workload is separated in eight consecutive phases. Each 
phase comprises 200 statements and favors statements on specific 
data sets, thus requiring a different set of indices for effective tun- 
ing. Adjacent phases overlap in the focused data sets and also differ 
in the relative frequency of updates and queries. (See |15| for fur- 
ther details on data and SQL statements.) The specific workload 
is a difficult use case for index tuning due to the mix of updates 
and queries and the alternation of phases. In fact, the DB2 index 



Competitor Techniques. We compare wfit empirically against 
two competitor algorithms. The first algorithm, termed BC, is an 



^The original algorithm was developed in the context of MS SQL 
Server. Some of its components do not have counterparts in DB2. 



advisor was unable to recommend a beneficial index configuration 
for the whole workload, even with an infinite storage budget for 
indices. (We obtained similar experimental results with workloads 
of lower query complexity.) 

Performance Metrics. We measure the performance of an on- 
line algorithm A using totWork{A, Q„, V) for the previously de- 
scribed workload and some feedback stream V. The definition of 
V depends on the experiment and is detailed when we present the 
results. As in previous studies on index tuning (5||6||15|, the total 
work metric is evaluated using the optimizer's cost model. The goal 
is to isolate the performance of A from any cost-estimation errors, 
e.g., due to insufficient data statistics or faulty cost models. 

In all experiments, the we measure the performance of A as 
totWork{oPT, Qn, V) / totWork{A, Q„, V), which indicates the 
performance of A relative to the optimal recommendations of OPT. 
We note that the OPT can have very different recommendation sched- 
ules for Qn and Q„+i respectively, whereas A's recommendation 
schedule for Q„+i is an extension of the schedule for Q„. 

We also report the overhead of algorithm A in terms of two com- 
ponents: the number of what-if optimization calls, and the remain- 
der of the overhead as absolute wall-clock time. The reason for this 
separation is that the efficiency of the what-if optimizer is some- 
what independent of the tuning algorithm. Indeed, techniques for 
very fast what-if optimization ||8J can reduce substantially the over- 
head of any tuning task. 

Generating the Fixed Stable Partition. As explained above, we 
choose a fixed stable partition {Ci, . . . , Ck} to be used by the 
competing algorithms. We developed an automated method to com- 
pute this partition in a way that captures the most relevant indices 
and interactions in the entire workload. Specifically, we first obtain 
a large set of interesting indices U by invoking DB2's index advisor 
on the read-only portion of the workload with an infinite space bud- 
get (as mentioned earlier, the index advisor would not recommend 
any indices to create for the entire workload). We then choose a 
subset C C and a partition of C, using an offline variation of the 
chooseCands algorithm. The only change to chooseCands is to 
compute an average of the benefit and degree of interaction over the 
entire workload (rather than a suffix), and use these measurements 
as the criteria for the top indices and stable partition. For the work- 
loads in our experiments, U contained roughly 300 indices, and the 
size of the stable partition depended on the parameter settings of 
WFIT. 

6.2 Results 

Baseline Performance. We begin with a baseline experiment where 
the stable partition is fixed and no feedback is provided (V = 0). 
In this setting, WFIT becomes equivalent to WFA+ (Section |4]l and 
the measured performance reflects the effectiveness of the index 
recommendation logic. It also becomes possible to make a mean- 
ingful comparison to BC, which does not support feedback. 

Figure [8] shows the normalized performance metrics for WFIT 
and BC. For WFIT we chart three curves that correspond to three 
different settings of 2000, 500, and 100 for the stateCnt parameter 
of the stable partition. A high value corresponds to a more detailed 
stable partition that provides more information to WFIT but also 
increases its overhead. (The complexity of WFIT grows quadrati- 
cally with stateCnt.) Figure[8]also includes a fourth curve labeled 
WFIT-IND, which corresponds to a variant of WFIT that considers 
all indices to be independent. In other words, this version of the al- 
gorithm assumes doiq{a, b) = for all indices and queries, which 
means that each index is in a separate singleton part. This version 
of WFIT would not be used in practice, but we show its performance 



in order to see the value of analyzing index interactions. 

As shown, the quality of recommendations degrades gracefully 
as stateCnt decreases from 2000 down to 100, with the over- 
all difference remaining small throughout. The drop in perfor- 
mance is more significant for WFIT-IND, where all index interac- 
tions are ignored. We performed experiments with higher settings 
of stateCnt, up to 10000, but we omit the results, as there was 
very little difference compared to stateCnt — 2000. Essentially, 
the results show that WFIT can generate effective recommendations 
as long as the stable partition captures the important interactions 
among the candidate indices. 

Another observation from Figure [8] is that WFIT's performance 
comes very close to the algorithm that has complete knowledge of 
the workload. The difference is less than 10% at the end, which is 
very significant if one considers the complex mix of updates and 
join queries in the workload. It is interesting to examine this em- 
pirical performance against the theoretical competitive ratio stated 
in Section |4] For this particular experiment, there are 8 indices in 
the biggest part of the stable partition and hence the performance 
of WFIT should always be within a factor of 2*"*"^ — 1 of optimal. 
As shown by the results, WFIT's performance can be much better 
compared to this worst-case bound. 

Finally, Figure [S] shows that WFIT outperforms BC by a signif- 
icant margin. The difference becomes substantial after the initial 
statements in the workload, and by the end WFIT (without the in- 
dependence assumption) attains >90% of the performance of OPT 
compared to 65% for BC. The difference shows that WFIT's prin- 
cipled handling of index interactions is more effective than the 
heuristics used by BC. In fact, the results show that even WFIT- 
IND outperforms BC on this workload. This could be due in part 
to the fact that our adaptation of BC is implemented outside the 
DBMS, and the original design of BC may be better suited for an 
internal implementation that is closely integrated with the query 
optimizer. 

Overhead. For the same experiment, the Java implementation of 
WFIT on top of DB2 required 300ms on average to analyze each 
query and generate the recommendations. This magnitude of over- 
head is acceptable if one considers the much higher query execution 
cost and the savings obtained from having the right indices materi- 
alized. Still, overhead can be reduced substantially with a careful 
implementation inside the DBMS, or by switching to a lower value 
for stateCnt. For instance, setting stateCnt = 100 will not affect 
significantly the quality of recommendations (see Figure [8]l but it 
can reduce the overhead by a factor of 25. A different solution is to 
do the analysis in a separate machine (e.g., the DBA's workstation) 
without any impact on normal query evaluation. 

Regarding the number of what-if optimizations, WFIT averaged 
between 5 and 100 calls per query close to the start and end of 
the experiment respectively. The number of what-if calls is directly 
correlated with the number of candidate indices that are mined from 
the workload. A different implementation of WFIT could constrain 
the latter, but the experimental results of Bruno and Nehme (8j sug- 
gest that it is possible to perform 100 what-if calls per query while 
keeping up with the flow of the workload. 

The Effect of Feedback. The next set of experiments evaluates 
WFIT's feedback mechanism (SectionjSTT}, one of the core features 
of the semi-automatic tuning paradigm. 

We examine the performance of WFIT for two contrasting models 
of DBA feedback. The first model, represented with a feedback 
input Vgood, represents "good" feedback where the DBA casts a 
positive (resp. negative) vote for index a at point n in the workload 
if OPT creates (resp. drops) a after analyzing query n. The idea is 
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to model a prescient DBA who can use votes to guide WFIT toward 
the optimal design. We also create a "bad" feedback input, denoted 
as Vbad, as the mirror image of good feedback, i.e., we replace the 
positive votes with negative votes and vice versa. 

Figure |9] shows the performance of WFIT for V — Vgood and 
V = Vbad. As a baseline, we include a run of WFIT without 
feedback, i.e., 1/ = 0. The results show that the feedback mech- 
anism works intuitively. The useful feedback improves the per- 
formance of the baseline and pushes it closer to the optimal al- 
gorithm. WFIT does not exactly match the performance of OPT, 
since the latter computes its recommendations using much more 
detailed information (recall that WFIT uses a fixed stable partition 
with stateCnt — 500). The bad feedback causes a degradation of 
performance, as expected, but WFIT is still able to output effective 
recommendations and remain above 90% of optimal by the end of 
the workload. The key point is that WFIT initially biases its recom- 
mendations according to the erroneous feedback, but it is able to 
recover based on the subsequent analysis of the workload. 

It is also interesting to examine the effect of feedback in the mod- 
ified version of WFIT which assumes all indices are independent. 
This experiment models an interesting scenario for the usefulness 
of semi-automatic tuning, as the assumption of index independence 
can introduce significant errors in WFIT's internal statistics on in- 
dex benefits, and hence DBA feedback can have a significant effect 
on the quality of the generated recommendations. Figure[TO]shows 
the result of providing feedback as Vgood for the WFIT-IND algo- 
rithm. (We omit results that combine WFIT-IND with the "adver- 
sarial" feedback Vbad, since such a scenario would stray too far 
from what would be seen in practice, and the results would have 
little meaning.) The results show that the DBA's feedback can still 
improve the quality of the recommendations significantly, despite 
the fact that WFIT has very inaccurate internal statistics. 

Delayed Feedback. The previous experiments assumed that the 
DBA accepts the recommendation of WFIT after each query. In 
contrast, the next experiments evaluate the effect of delayed feed- 



back, which is what we expect to see in practice. We model this 
scenario with a feedback input Vt, where the DBA requests and 
accepts the current recommendation of WFIT every T queries. This 
feedback renews the "lease" of the current recommendation, which 
in turn delays WFIT from switching to a potentially better recom- 
mendation. Hence, some degradation in performance is possible. 

The results of this experiment are shown in Figure [TT] The first 
curve shows the performance for T — 1, which grants full auton- 
omy to WFIT. The other curves show the result of increasing the 
delay T to 25, 50, and 75. There is clearly a loss in overall perfor- 
mance when the responses of the DBA are delayed. At the end of 
the workload, the performance with T > 1 is around 85% of op- 
timal, which is below the 95% level achieved by wfit without the 
lag. A close examination of the results reveals that most indices are 
beneficial only for short windows of the workload, due to interven- 
ing updates that make indices expensive to maintain. This aspect of 
the workload makes the delayed responses particularly detrimental, 
and reflects our choice of this workload as a "stress test" for WFIT. 
However, it is important to observe that the performance does not 
continue to degrade as the length of the lag increases. We limited 
the lag to 75 queries in order to avoid a lag that spanned a large 
portion of the phase length of 200 queries. In general, the results 
suggest that semi-automatic interface can provide robust recom- 
mendations even when the lag is significant compared to the phase 
length. 

Automatic Maintenance of Stable Partition. The final set of ex- 
periments examines the performance of WFIT when chooseCands 
is used to maintain the stable partition automatically, as described 
in Section [5!2| In this case, the stable partition may change over 
time, which causes repartition to be invoked. We compare this 
approach to the variation of WFIT with a fixed stable partition. 

Figure [12] shows the performance of WFIT with a fixed stable 
partition and with automatic maintenance of candidates, labeled 
FIXED and AUTO, respectively. We see an overall improvement 
in the performance using chooseCands to maintain the indices and 



interactions on-the-fly. Overall, chooseCands mined about 300 
candidate indices from the workload, and changed the stable parti- 
tion 147 times over the course of the experiment (although many of 
the calls to repartition only made minor changes to the modeled 
interactions). The observed performance clearly validates the abil- 
ity of repartition to update the internal state of WFIT in a meaning- 
ful way. We also observe that the performance slightly exceeds OPT 
in the earlier queries, which are mostly read-only statements. This 
is due to the fact that the automatic maintenance of the stable parti- 
tion allows WFIT to specialize the choice of indices for each phase, 
whereas OPT is limited to one set of candidates for the workload. 

7 Conclusions 

We introduced the novel paradigm of semi-automatic index tun- 
ing, and its realization in the WFIT algorithm. WFIT leverages and 
extends principled methods from online computation. Experimen- 
tal results validate its numerous advantages over existing techniques, 
and the feasibility of semi-automatic tuning in practice. 
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APPENDIX 

A Competitive Analysis 

This section provide s pro ofs of our results on the competitive 
ratio of WFA (Theorem 



4.1 



and WFA^ (Theorem 4.3 1. The proof 
starts with two technical lemmas that lead to a central result given 
in Theorem |A.I| This theorem leads immediately to Theorem |4.I[ 
and it is also used to prove Theorem |4.3| with a bit more machinery. 

We begin with notation. Fix the workload Q of N statements 
and denote the i-th query as qi. In the context of this analysis, the 
algorithms choose recommendations that are subsets of a fixed set 
of candidate indices C. Prior to observing any queries, the materi- 
alized set of indices is some set 5*0 C C. 

We introduce a formal expression for the transition cost S{X, Y) : 

S{X,Y)= J2 E '^"W 

aeY-X aeX~Y 

where 5'^ (a) and 5" (a) denote respectively the cost of creating 
and dropping index a. We use the shorthand notation 



WFA„ 



totWork{WFA, i 



for the total work of WFA on a prefix Q„ of the workload. We 
define the shorthand WFA^ and OPT„ similarly. 

For each query qi we fix a set 5** that minimizes the cost of qi : 

S* = arg min cost(qi, X). 

The sum of these values over a prefix of the workload is denoted 

BASE„ = YJi^i cost{qi,S*). 

Our analysis makes frequent use of these quantities as a simple 
lower bound on the query processing cost that must be paid by any 
algorithm. The following result shows that the minimum query cost 
also bounds the amount that the work function for an individual 
state increases after each statement. 

Lemma A.I. Wi+i{S) > Wi{S) + cost{qi+i, 8*^1) for all i>0. 
Proof: For the case i = 0, we apply the triangle inequality of S: 



mm{wo{X) + cost{qi,X) + S{X, S)} 
mm{S{So,X) + cost{qi, X) + S{X, S)} 



> 5(So, S) + rain cost{q\, X) 
= Wq{S) + cost{qi, Si). 

For the case i > 1, note that there exist two index configurations 
Pi, Pi-i that satisfy the following equations: 

Wi+i{S) = w,{Pi) + cost{q,+ i,Pi) + S(Pi,S) 
Wi{Pi) = w^^i{Pi-i) + cost{qi,Pi-i) + S{Pi-i,Pi) 

In other words, the final steps in the path corresponding to Wi+i (S) 
are Pi-i — > Pi — >■ S. The path corresponding to 'Wi{S) may have 
a different configuration as a predecessor to S, but we can use the 
path that passes through Pi-i to bound the value: 

w,{S) < w,^i{P^i) + cost{q„P^-i) + 5{P^-i,S) 
= w,{P)-5{P,-i,P,) + 5{P^-i,S) 



< w,{P) + S{P,S) 



The triangle inequality yields the third step. It follows that 

'Wi+i{S) > WiiS) + cost{qi+i, Pi) > uii(S) + cost{qi+i,S*+i) 
as desired. 

We next give a result that shows the transition cost of a cyclic 
sequence of configurations does not change if we reverse the cycle. 

Lemma A.2. Consider the sequence of index configurations 
So, Si, . . . , Sn, So Q C. The following identity holds: 

n n 

^ <5(S'.-i, 50 + 5{S„, So) = "^'-i) + ^(^0, S„) 

i = l i = l 

Proof: By induction on n. The base cases n — 0,1 are triv- 
ial, so we consider the interesting base case n — 2. For the cy- 
cle So, Si, S2, So, the total transition cost is equal to the sum of 
S'^{a) + 5~ (a) over all indices a that occur in exactly one or two 
of the sets 5o, Si, S2. This can be checked by enumerating the 
possible sets that such indices can occur in. By symmetry, the tran- 
sitions on the reverse cycle So, S2, Si, So have the same cost. 

Now consider the inductive case n > 3. By an application of the 
inductive hypothesis lo So, Si, . . . , S„-i, So, 

n n — 1 

^<5(5,_i,5,) + 5(5„,So) = ^5(5„5,_i)-5(S„_i,5o) 

i-l i = l 

+ (5(So, 5„_i) + <5{5„_i, S„) 
+ S{S„,So) 

The r.h.s. contains the cost of the cycle So, S„_i, S„, So. Since 
we assumed n > 3, we may apply the inductive hypothesis once 
more, and replace these terms with the cost of the reverse cycle. 
The lemma follows from this substitution. | 

We can now prove the central result of this section. The theorem 
shows a bound on the cost of WFA that is strictly stronger than the 
competitive ratio of Theorem |4.1| when the minimum query costs 
cost{qi, S*) are significant compared to the cost of the optimal 
schedule. The stronger statement is needed to prove Theorem |4. 3 1 

Theorem A.l. The total work of WFA satisfies 

WFAjv — BASEat < (2''^' + "^ — l)(OPTjv — BASEiv) + a 
where a does not depend on the workload Q. 

Proof: We follow the overall strategy of the proof of Lemma 9.3 
in jSj, which is an analogous result for task systems with symmet- 
ric transition costs. Our setting differs since S is not symmetric. 
The original proof also does not consider the effect of the terms 
cost{qi, S*) that we account for in the theorem. 

We start with notation borrowed from |3 1: 

fi = max{S{X,Y)\X,Y <ZC} 
Si = configuration recommended by WFA for qi 

Bi = Es + Es#s, ('S') 

Our eventual goal is to derive separate bounds for WFA n and OPT m 
with respect to Bn, and then combine these bounds. We first ob- 
serve that 

B,+i-B, = w,+i{S{)-w,(S,+i) + 2- ^ K+i(5)-«),(5)) 

+ Wi+i{Si+i) - Wi{Si+i) + Wi+i{Si) - Wi(Si) 
> Wi+i{Si) - Wi{Si+i) 



by Lemma [ATT| We can show that 

Wi+i{Si) — Wi{Si+i) > S{Si+i,Si) + cost{qi+i,Si+i) 

using the same reasoning as js] (see the original proof for detail^. 
Hence, 

Bi+i-Bi > 5{Si+i,S,) + cost{qi+i,Si+i) 

+ (2lCI+i„2)-cost(g,+i,S*+i). 

Summing these inequalities for < i < A'^ — 1 yields 

AT 

Bn -Bo > ^ S{S„ S,-i) + cost{q„S,) 



(2 



C| + l 



2) • BASEjv. 



The first line of the r.h.s. is similar to the total work of WFA ex- 
cept that the transition costs are reversed. We can remedy this via 
Lemma|A.2| which leads to 



Bo > ^5(Si-i,S0 + cost(g,,S.: 



B, 



+ S{Sn, So) — 5{So, Sjv) 

+ (2l'^l + ^ - 2) • BASEat 
= WFAat - (2'*^' + ^ - 2) ■ BASEat 

+ S{Sn, So) — S{So, Sn) 
Finally, we can bound WFA at — BASEjv by 

WFAat — BASEiv < Sat + (2''^' + "^ — 1) ■ BASEat 

— Bo — S{Sn, So) + 5{So, Sn) 
To complete the proof, we note that 

Bn < (2l^l+^ - 1) • OPT AT + (2'^'+' - 2)n 
as shown in |3|. | 



Proof of Theorem |4.1| Rearranging the terms in Theorem |A.l| we 

have 



WFAat < (2'^^' + ^ - l)OPTAr - (2'^' + ' - 2)BASEat + Q 

where a does not depend on the work load Q. Since (2ICI+1 _2) 
and BASEat are nonnegative. Theorem 4.1 follows. | 



JCI + l 



(2 



C|+l , 



2) ■ cost{qi+i,Si^i) 



Proof of Theorem |4.3| We now show the competitive ratio of 
WFA+ using a fixed stable partition {Ci, . . . , Ck} of the candi- 
date indices C. Our strategy is to use Theorem |A.1| to analyze the 
recommendations chosen by WFA within each part Ck ■ 

We first extend some of our previous notation to describe the 
behavior of WFA in an individual part Ck- We use WFa'''' to repre- 
sent an instance of WFA that selects recommendations from the part 
Ck only. Similarly, opt'*' is the idealized algorithm that chooses 
the optimal recommendations from Ck with advance knowledge of 
the workload Q. Our shorthand for total work extends naturally, 
e.g., WFAi'"' denotes the total work of the path that implements the 

"^The inequality from fs | reverses the arguments to 5. In the original 
proof, the distinction is not important because the transitions are 
assumed to be symmetric. In order to prove the competitive ratio in 
our setting, the arguments to S are reversed in the criteria that WFA 
uses to select the next recommendation. This results in the slightly 
different inequality. 



recommendations of WFA*^*' for the first n queries. We denote 

Si — recommendation of WFA^ for qi 
S^''^ = SiflCk = recommendation of WFA*'°' for qt 
Oi — recommendation of OPT for qi 

It follows easily from \2A\ that OiOCk, ■ ■ ■ , On n C'k is an optimal 
path within the part C'k of the stable partition. A similar fact is: 



S* nCk = arg min cost{qi,X) 

XCCi- 



In other words, the minimum query processing cost using configu- 
rations within C'k is achieved by 5** H Ck ■ Thus we use the notation 

baseL*' = Er=i cost{q,, S* n Ck) 
We can also break down the transition cost based on the partition: 

S{x, Y) = Y^S{xnCk,YnCk) 

k 

We need to express the total work of WFA+ an d OP T w.r.t. the 

Note that 



A.l 



lower bounds BASE^' in order to apply Theorem 
OPT's recommendations Oo, . . . , On obey the following identity: 

cost{qi, Oi) — cost{qi, S*) 

= benefit^. (S* , 0) - benefit {Oi, 0) 

= benefit (S* n Cfc , 0) - benefit (O,- n Ck , 0) 

k 

= ^ cost{qi, Oi n Ck) — cost{qi, S* R Ck) 



Now we rewrite the total work of OPT, offset by BASEjv: 

OPTjv - BASEjv 

JV 

= ^<5(Oi_i,Oi) + cost{qi,Oi) - cost(qi,S*) 

= ^<5{o,_i,Oi) + ^cost((ji,OinCfc)- cost{qi,s*i r\Ck) 

i k 

= ^^(5(Oi_i nCfe,o, nCfe) + cost{qi,o^nCk) 

k i 

i 

= E°PT<v''-BASE« 



Applying the same steps to the sequence So, ... , Sn, we can de- 
rive an analogous bound for the cost of WFA+ : 



WFA^Y — BASEjv = J2k WFA^' — BASE 



(fe) 
JV • 



Now by Theorem 



A.l 



we know that WFA\,' — BASE)^^ is bounded 



^jv BASEjY 

above by (2l^'=l+^ - 1)(0PT^'' - BASE^'') + Ok where ak does 
not depend on the workload. Then we obviously have 



WFA^'' - base);' < (2'=— + ^ - l)(OPT^'' - BASE^'O + ak 
where, as in the statement of Theorem |431 Cmax is defined as the 



maximum of |Cfe| for all k. We apply this as follows: 

WFa]^ = BASEjv + ^ WFA^^' - BASE^' 



< BASEjv + (2'=™- + l - l)^{OPT^-' - BASE*^*^ + E 

fe k 

= BASEjv + (2'^»"'»'' + -'^ - l)(OPTjv - BASEjv) + ^ <^fc 

fc 

= (2'=»='^ + l - l)OPTjv - (2'="""' + l - 2)BASEjv + ^«fe 

k 

< (2-max + l_l)opTjv+^afe 



This proves that the competitive ratio of WFA is 2 



B Proof of Theorem 4.2 



+ OCmax + l 



1.1 



Theorem 4.2 states that if WFA^ uses any fixed stable partition 
{Ci, . . . , C'k} of the indices C, it can generate the same recom- 
mendations as the naive application of WFA that jointly tracks all 
subsets of C. Before the main proof, we give one preliminary result 
that describes the relationship between the global work function 
and the work functions that WFA^ maintains for each part. 

Lemma B . 1 . Let w„ be the work function for the workload Q„ 

(k) 

and the indices C. Let yvfi he the work function values calculated 
by WFA^ for Ck after observing Qn. For any S C, 

n 

w„iS) = wi'^ [S n Ck] - (K -1)Y, cost{q^, 0). 

fe i=l 

Proof: By induction on n. For the base case n = 0, the sum is 
empty, and the theorem follows from the identity 

= Efe'5(^nCfe,5nCfe) 

mentioned in earlier sections. Specifically, we have 

wo{s) = 5{Sq, s) = Efe 5(SonCfe, snCfe) = Efe W(<*^(S'nCfe). 

To prove the inductive case n > 1, we observe the following 
identity that follows easily from \2.iy . for all X C C, 

cost{q„,X) = J2kC0stiqn,XnCk)-{K-l)cost{q„,t!)) 

Recall the definition of the work function 

w„{S) = mm{K;„_i(X) + cost{q„,X) + S{X, S)} 

If we apply the inductive hypothesis to and also decom- 

pose the terms cost{qn, X) andS{X, S) as shown above, the result 
simplifies to 

■w„{S) = mm{J2^^v/l^l-^(XnCk) + cx>st{q„, X Ck) 

"^-"^ + 5(xnCfc,snCfe)} 

-(i^-l)E-U cost{q,,%). 

The terms in the sum over k each depend on a disjoint part Ck, so 
the summation can be pulled out of the min operation to yield 

^^{S) = Efe min {yvl%{Xk) + cost{qr„Xk) + S{Xk,SnCk)} 

= Efe'«'i'='[snCfe]-(ii--i)Er=iCO5tfe,0). I 

In order to prove the equivalence between WFA^ and WFA, we 
must resolve the fact that the selection criteria of WFA are not de- 
terministic: if more than one configuration satisfies the criteria, the 



pseudocode does not specify which configuration is chosen. Thus, 
we assume a simple tie-breaker based on lexicographic ordering, as 
follows. Let {ai, . . . , a|c|} denote the indices in C. If X,Y (Z C 
and X Y, consider the minimum value of d where X and Y dif- 
fer on ad, meaning that ad is in the symmetric difference X QY. 
The lexicographic tie-breaking rule prefers X to y iff G X. 

We consider a workload Qn of length n. Let So be the initial 
configuration. Si , . . . , S„ be the recommendations of WFA, and 
Si , . . . , S^ be the rec omm endations of WFA^ within each part Ck ■ 
The claim of Theorem ^ 



4.2 



can be stated as S„ = Ufe for w > 0. 
The proof proceeds by induction on n. The base case n = is 
trivial, as both algorithms start with the same state. 

Consider the inductive case n > 1. Assume for contradiction 
that Sn 7^ Ufc "^n- Take the minimum d where ad € S„ IJfe 5'^;. 
Let Ce be the part that contains ad, implying either ad € — S„ 
or ttd € Sn — S^. We first consider the case ad G S^ — S„. Let Sn 
denote (Sn — Ct) U S^ which is the result of modifying S„ to be 
consistent with WFA+'s choice within Ce. We immediately observe 
that the lexicographic tie-breaker prefers Sn to Sn. The set Sn 
also satisfies WFA's explicit tie-breaking constraint Sn £ p[Sn], by 
virtue of the fact that both Sn and Sn satisfy the constraint. Hence, 
the only possible reason that WFA recommends Sn instead of Sn 
must be that score{Sn) < score{Sn), i.e., 

Wn(Sn) + 5(Sn, Sn-l) < Wn{Sn) + 5{S„, Sn-l). 



We may use Lemma |BTT] to decompose both sides of the inequality 
according to the stable partition. Since Sn and Sn agree on all 
indices outside of Ce, we may cancel terms to yield: 

\y'n\Sn n Ce) + 5{Sn n Ce, 5n-i n Ce) 

<y^i^''{Si) + 5{Si,Sn-inCe). 

I.e., Sn n Ce has a better score than S^, contradicting the fact that 
WFA+ recommends Sn within Ce. 

The proof when ad € Sn — Sn is completely symmetric: we 
contradict WFA's choice of Sn by showing Sn has a lower score. | 



